hanzoai/datastore
2
0
Fork 0
mirror of https://github.com/hanzoai/datastore.git synced 2026-09-26 13:55:07 +00:00
Code Issues Packages Projects Releases Wiki Activity
73,726 Commits 5 Branches 2,005 Tags
9d69786a6b95d0d5eb64a53ccc41cdf46f2785dd
Commit Graph
308 Commits
Author SHA1 Message Date
Azat Khuzhin 9d69786a6b Increase listen_backlog by default (to match default in newer linux kernel) 
It was 64 for a long time, and even linux kernel never has such a small
limit, it had 128 (net.core.somaxconn sysctl).

But recently, in 5.4, the default value had been increased even in
linux kernel, to 4096 [1].

  [1]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=19f92a030ca6d772ab44b22ee6a01378a8cb32d4

Let's increase it in ClickHouse too.

Also note, that I've looked through some instances with
clickhouse-server only, and TcpExtListenOverflows was non zero (`nstat
-za`), in other words backlog of the listen socket indeed overflowed
there.

So it is better to increase the default to move the problem to
clickhouse-server itself (yes you will unlikely have 4K new incomming
connections at one time, with accept thread do not accept them all, but
still seems that it is possible, maybe due to some locks or something
else).
 2021-10-01 21:03:59 +03:00
Nikita MikhaylovandGitHub 9756b8dc33 Added an ability to execute more merges and mutations than threads, added new settings (#29140) 2021-10-01 00:26:24 +03:00
Raúl Marín 462d36195c Allow reloading max_concurrent_queries without restart 2021-09-27 12:32:35 +02:00
Anton Popov 83e45adad6 Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-20 20:04:50 +03:00
tavplubixandGitHub 922cf7ee20 Merge pull request #28373 from ClickHouse/tables_topsort 
Resolve table dependencies on metadata loading
 2021-09-20 14:46:47 +03:00
Maksim Kita 85a4d4bb50 Added user defined executable functions to system.functions 2021-09-17 18:43:00 +03:00
Maksim Kita ca395e9678 Fixed tests 2021-09-17 18:43:00 +03:00
Maksim Kita 55492cc9bf Fixed build 2021-09-17 18:43:00 +03:00
Maksim KitaandMaksim Kita de20e04dfd Added executable user defined functions 2021-09-17 18:42:59 +03:00
Anton Popov 99175f7acc minor enhancements in async inserts 2021-09-16 20:55:34 +03:00
Anton Popov fc17936c12 Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-14 23:02:30 +03:00
Alexander Tokmakov 8ac19caca9 Merge branch 'master' into tables_topsort 2021-09-13 16:58:39 +03:00
Filatenkov ArturandGitHub c23fe5baf6 Improve codec for encr 19896 (#27476) 
* change syntax of encrypted command

* commit all encrypted changes

* correct encryption

* correct config for test

* add tests and correct code style and typos

* correct test

* fix unbundled build

* add log warning messages

* improve code according to review comments

* correct nonce

* correct errors found by fuzzing

* improve codec AES_128_GCM_SIV. Add AES_256_GCM_SIV. Add sections for last in tests. Improve documentation

* Update CompressionCodecEncrypted.h

* Update 01683_codec_encrypted.sql

* correct compression factory after changes in master

* correct behavior with wrong key in data

* correct fuzzer

* add connection for fuzzer with fix for compression_encrypted

* refactor code

* add load from config with throwing errors on server start

* fix typos and check style

* Update Server.cpp

* correct loading and reading

* refactor code. fix uninitialized value

* refactor code

* move defines from server to cpp file

* correct build

* remove repeated code

* correct namespace

* fix code style
 2021-09-13 11:25:36 +03:00
Anton Popov 5cff615eca Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-10 13:40:48 +03:00
Alexander Tokmakov 3f8fb1e562 Merge branch 'master' into tables_topsort 2021-09-10 13:16:41 +03:00
Maksim Kita e244239238 Lower compiled_expression_cache_size to 128MB 2021-09-09 19:35:06 +03:00
Anton Popov 106566e701 fix server 2021-09-09 18:01:39 +03:00
Anton Popov 8e5b3b2f6c Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-09 15:12:34 +03:00
tavplubixGitHubDamir PetrovDamir Petrovmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		341a6c51d6 Merging #24866 (#28691) 
* Add StorageSystemISTables.cpp/.h

* Another attempt

* Columns and Views

* Add information schema db and fix information schema 'tables' table

* fix build

* remove copy-paste, add views to system tables

* add test

* fix

* fix_tests

Co-authored-by: Damir Petrov <[email protected]>
Co-authored-by: Damir Petrov <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2021-09-09 12:37:51 +03:00
Nikita MikhaylovandGitHub 4db5062d6b Merge pull request #28374 from nikitamikhaylov/global-merge-executor 
Introduced global executor for background MergeTree-related operations
 2021-09-09 11:30:21 +03:00
alexey-milovidovandGitHub 67c16b3a4d Merge pull request #28761 from azat/uncaught-exception-fix 
Fix uncaught exception during server termination
 2021-09-09 08:57:28 +03:00
Azat Khuzhin df414ae71c Fix uncaught exception during server termination 
Example of a stacktrace:

<details>

```
[ 47463 ] {} <Trace> BaseDaemon: Received signal 15
[ 47463 ] {} <Information> Application: Received termination signal (Terminated)
[ 47462 ] {} <Debug> Application: Received termination signal.
[ 47462 ] {} <Debug> Application: Waiting for current connections to close.
[ 47463 ] {} <Trace> BaseDaemon: Received signal 15
[ 47463 ] {} <Information> Application: Received termination signal (Terminated)
...
[ 47463 ] {} <Trace> BaseDaemon: Received signal -1
[ 47463 ] {} <Fatal> BaseDaemon: (version 21.9.1.1, build id: 63945F58FC2C28ED) (from thread 47462) Terminate called for uncaught exception:
[ 47463 ] {} <Fatal> BaseDaemon: Code: 210. DB::NetException: Connection reset by peer, while writing to socket (10.7.141.42:9000). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):
[ 47463 ] {} <Fatal> BaseDaemon:
[ 47463 ] {} <Fatal> BaseDaemon: 0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0x94ca99a in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 47463 ] {} <Fatal> BaseDaemon: 1. DB::WriteBufferFromPocoSocket::nextImpl() @ 0x10676a3b in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 47463 ] {} <Fatal> BaseDaemon: 2. DB::Connection::sendCancel() @ 0x11554701 in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 47463 ] {} <Fatal> BaseDaemon: 3. DB::MultiplexedConnections::sendCancel() @ 0x1157e766 in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 47463 ] {} <Fatal> BaseDaemon: 4. DB::RemoteQueryExecutor::tryCancel(char const*, std::__1::unique_ptr<DB::RemoteQueryExecutorReadContext, std::__1::default_delete<DB::RemoteQueryExecutorReadContext> >*) @ 0x10392000 in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 47463 ] {} <Fatal> BaseDaemon: 5. DB::PipelineExecutor::cancel() @ 0x11697ffe in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 47463 ] {} <Fatal> BaseDaemon: 6. DB::QueryStatus::cancelQuery(bool) @ 0x10c19fc8 in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 47463 ] {} <Fatal> BaseDaemon: 7. DB::ProcessList::killAllQueries() @ 0x10c1a6ae in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 47463 ] {} <Fatal> BaseDaemon: 8. basic_scope_guard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_18>::~basic_scope_guard() @ 0x95587ad in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 47463 ] {} <Fatal> BaseDaemon: 9. DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x95528a2 in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 47463 ] {} <Fatal> BaseDaemon: 10. Poco::Util::Application::run() @ 0x141e85a3 in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 47463 ] {} <Fatal> BaseDaemon: 11. DB::Server::run() @ 0x9541dac in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 47463 ] {} <Fatal> BaseDaemon: 12. mainEntryClickHouseServer(int, char**) @ 0x9540153 in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 47463 ] {} <Fatal> BaseDaemon: 13. main @ 0x94c569e in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 47463 ] {} <Fatal> BaseDaemon: 14. __libc_start_main @ 0x26d0a in /usr/lib/x86_64-linux-gnu/libc-2.31.so
[ 47463 ] {} <Fatal> BaseDaemon: 15. _start @ 0x9490a2a in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 47463 ] {} <Fatal> BaseDaemon:  (version 21.9.1.1)
[ 47463 ] {} <Trace> BaseDaemon: Received signal 6
[ 11858 ] {} <Fatal> BaseDaemon: ########################################
[ 11858 ] {} <Fatal> BaseDaemon: (version 21.9.1.1, build id: 63945F58FC2C28ED) (from thread 47462) (no query) Received signal Aborted (6)
[ 11858 ] {} <Fatal> BaseDaemon:
[ 11858 ] {} <Fatal> BaseDaemon: Stack trace: 0x7ff04c196ce1 0x7ff04c180537 0xff91f28 0x163304e3 0x1633044c 0x94c60cb 0x10c1a135 0x10c1a6ae 0x95587ad 0x95528a2 0x141e85a3 0x9541dac 0x9540153 0x94c569e 0x7ff04c181d0a 0x9490a2a
[ 11858 ] {} <Fatal> BaseDaemon: 1. raise @ 0x3bce1 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
[ 11858 ] {} <Fatal> BaseDaemon: 2. abort @ 0x25537 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
[ 11858 ] {} <Fatal> BaseDaemon: 3. terminate_handler() @ 0xff91f28 in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 11858 ] {} <Fatal> BaseDaemon: 4. std::__terminate(void (*)()) @ 0x163304e3 in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 11858 ] {} <Fatal> BaseDaemon: 5. std::terminate() @ 0x1633044c in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 11858 ] {} <Fatal> BaseDaemon: 6. ? @ 0x94c60cb in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 11858 ] {} <Fatal> BaseDaemon: 7. ? @ 0x10c1a135 in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 11858 ] {} <Fatal> BaseDaemon: 8. DB::ProcessList::killAllQueries() @ 0x10c1a6ae in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 11858 ] {} <Fatal> BaseDaemon: 9. basic_scope_guard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_18>::~basic_scope_guard() @ 0x95587ad in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 11858 ] {} <Fatal> BaseDaemon: 10. DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x95528a2 in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 11858 ] {} <Fatal> BaseDaemon: 11. Poco::Util::Application::run() @ 0x141e85a3 in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 11858 ] {} <Fatal> BaseDaemon: 12. DB::Server::run() @ 0x9541dac in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 11858 ] {} <Fatal> BaseDaemon: 13. mainEntryClickHouseServer(int, char**) @ 0x9540153 in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 11858 ] {} <Fatal> BaseDaemon: 14. main @ 0x94c569e in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
[ 11858 ] {} <Fatal> BaseDaemon: 15. __libc_start_main @ 0x26d0a in /usr/lib/x86_64-linux-gnu/libc-2.31.so
[ 11858 ] {} <Fatal> BaseDaemon: 16. _start @ 0x9490a2a in /usr/lib/debug/.build-id/63/945f58fc2c28ed.debug
```

</details>
 2021-09-08 21:10:34 +03:00
mergify[bot]andGitHub 64fb384656 Merge branch 'master' into global-merge-executor 2021-09-08 13:15:38 +00:00
mergify[bot]andGitHub 9f0c50335f Merge branch 'master' into tables_topsort 2021-09-08 10:55:11 +00:00
Vitaly BaranovandGitHub 70c6623036 Merge branch 'master' into governance/session_log 2021-09-07 10:12:54 +03:00
Vitaly Baranov bcc31f1f3e Remove unnecessary changes. 2021-09-07 01:37:28 +03:00
Nikita Mikhaylov 7c39f84a5e Merge upstream/master into global-merge-executor (using imerge) 2021-09-06 15:37:18 +00:00
Azat Khuzhin bf6ba796f8 Fix UUID overlap in DROP TABLE for internal DDL from MaterializeMySQL 
This will fix race with DatabaseCatalog::loadMarkedAsDroppedTables(),
since MaterializeMySQL, and MaterializedMySQLSyncThread in background,
will be started earlier then
DatabaseCatalog::loadMarkedAsDroppedTables() and will move those tables
to metadata_dropped, and after loadMarkedAsDroppedTables() will start
and try to load partially dropped tables and will hit UUID overlap:

    12:02:51.536783 [ 3026034 ] {} <Information> Application: starting up
    12:02:53.019282 [ 3026034 ] {} <Information> DatabaseMaterializeMySQL<Atomic> (mysql): Total 9 tables and 0 dictionaries.
    12:02:53.041699 [ 3026200 ] {} <Debug> mysql.data (7143b65f-6982-4600-b143-b65f6982e600): Loading data parts
    12:02:53.041740 [ 3026200 ] {} <Debug> mysql.data (7143b65f-6982-4600-b143-b65f6982e600): There are no data parts
    12:02:53.620382 [ 3026034 ] {} <Information> DatabaseMaterializeMySQL<Atomic> (mysql): Starting up tables.
    12:03:00.669730 [ 3026183 ] {} <Debug> executeQuery: (internal) /*Materialize MySQL step 1: execute MySQL DDL for dump data*/ DROP TABLE mysql.data
    12:03:00.741894 [ 3026269 ] {} <Information> DatabaseCatalog: Trying load partially dropped table mysql.data (7143b65f-6982-4600-b143-b65f6982e600) from /var/lib/clickhouse/metadata_dropped/mysql.data.7143b65f-6982-4600-b143-b65f6982e600.sql
    12:03:00.742582 [ 3026269 ] {} <Debug> mysql.data (7143b65f-6982-4600-b143-b65f6982e600): Loading data parts
    12:03:00.742650 [ 3026269 ] {} <Debug> mysql.data (7143b65f-6982-4600-b143-b65f6982e600): There are no data parts
    12:03:00.773137 [ 3026034 ] {} <Error> Application: Caught exception while loading metadata: Code: 57, e.displayText() = DB::Exception: Mapping for table with UUID=7143b65f-6982-4600-b143-b65f6982e600 already exists. It happened due to UUID collision, most likely because some not random UUIDs were manually specified in CREATE queries., Stack trace (when copying this message, always include the lines below):
    12:03:01.224557 [ 3026034 ] {} <Error> Application: DB::Exception: Mapping for table with UUID=7143b65f-6982-4600-b143-b65f6982e600 already exists. It happened due to UUID collision, most likely because some not random UUIDs were manually specified in CREATE queries.

Cc: @zhang2014
 2021-09-04 10:41:28 +03:00
Anton Popov 7947e34a27 revert accidental changes 2021-09-03 22:21:01 +03:00
Alexander Tokmakov 024a24aaf7 better code, moar logging 2021-09-02 12:19:37 +03:00
Anton Popov 18dd1345fa return accidentally removed code 2021-09-02 02:46:23 +03:00
Anton Popov 5e421ab272 Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-02 02:42:50 +03:00
Vasily Nemkov 109d2f63d0 Fixed tests and minor style issues 2021-08-31 15:39:15 +03:00
Nikita Mikhaylov f8d4f04294 Merge upstream/master into global-merge-executor (using imerge) 2021-08-31 11:52:11 +00:00
Maksim KitaandGitHub 4ebd0ae941 Merge pull request #28102 from ClickHouse/executable-table-function 
Merging #23192
 2021-08-31 12:00:50 +03:00
Nikita Mikhaylov c4416906c8 done 2021-08-30 19:37:03 +00:00
Vasily Nemkov c902afddde Added system.session_log table 
Which logs all the info about LogIn, LogOut and LogIn Failure events.
Additional info that is logged:
- User name
- event type (LogIn, LogOut, LoginFailure)
- Event date\time\time with microseconds
- authentication type (same as for IDENTIFIED BY of CREATE USER statement)
- array of active settings profiles upon login
- array of active roles upon login
- array of changed settings with corresponding values
- client address and port
- interface (TCP\HTTP\MySQL\PostgreSQL, etc.)
- client info (name, version info)
- optional LoginFailure reason text message.

Added some tests to verify that events are properly saved with all necessary info via following interfaces:
- TCP
- HTTP
- MySQL

Known limitations
- Not tested against named HTTP sessions, PostgreSQL and gRPC, hence those are not guaranteed to work 100% properly.
 2021-08-30 18:28:28 +03:00
Anton Popov e8ac8e3454 execute asynchronous inserts separatly for each client 2021-08-27 06:00:12 +03:00
Amos Bird 0169fce78e Projection bug fixes and refactoring. 2021-08-26 19:09:31 +08:00
alesapinandGitHub 47b995467e Merge pull request #28080 from ClickHouse/some_renames_in_keeper 
Clickhouse-keeper: renames and comments
 2021-08-26 10:16:21 +03:00
Anton Popov 6d3274c22c Merge remote-tracking branch 'upstream/master' into HEAD 2021-08-26 03:31:46 +03:00
Maksim Kita 6ccdde4a88 Updated TableFunctionExecutable 2021-08-25 22:30:22 +03:00
Maksim KitaandGitHub dc852568c0 Merge pull request #27796 from ClickHouse/create-user-defined-lambda-function 
Merging #23978
 2021-08-25 10:08:20 +03:00
Maksim Kita 82ba24cd10 Merge branch 'master' into create-user-defined-lambda-function 2021-08-24 19:12:32 +03:00
Nicolae Vartolomei abc484e89b Destroy main_config_reloader before shared context. 
This tries to fix crash reported in a comment
https://github.com/ClickHouse/ClickHouse/pull/24404#issuecomment-847012002.
 2021-08-24 17:06:17 +01:00
alesapin 95cf0634dc Clickhouse-keeper: renames and comments 2021-08-24 15:30:31 +03:00
Vitaly Baranov 63e8bc1f20 Added new commands BACKUP and RESTORE. 2021-08-23 23:41:45 +03:00
Maksim Kita 2a6aa50d49 Merge branch 'master' into create-user-defined-lambda-function 2021-08-23 11:08:49 +03:00
alexey-milovidovandGitHub e620ea15bd Merge pull request #27900 from amosbird/newf 
getServerPort function
 2021-08-21 22:40:25 +03:00
Amos Bird 997acdc39e getPort function 2021-08-21 20:33:36 +08:00